Appendix G: The Message Understanding Conference Scoring Software User's Manual

Named Entity Scoring for Speech Input

This paper describcs a new scoring algorithm that supports comparison of linguistically annotated &tta from noisy sources. The new algorithm generalizes the Message Understanding Conference (MUC) Named Entity scoring algorithm, using a comparison based on explicit alignment of the underlying texts, followed by a scoring phase. The scoring procedure maps corresponding tagged regions and compares...

Appendix E: MUC-7 Named Entity Task Definition (version 3.5)

Description of the UMass system as used for MUC-6

Information extraction research at the University of Massachusetts is based on portable, trainable language processing components. Some components are more effective than others, some have been under development longer than others, but in all cases, we are working to eliminate manual knowledge engineering. Although UMass has participated in previous MUC evaluations, all of our information extra...

MUC/MET Evaluation Trends

During the course of the Tipster Program, evaluation methodology for information extraction developed as the technology progressed. Multiple task levels and multiple languages were successful targets of information extraction. Automated scoring and statistical significance algorithms were developed for use in scoring systems and for interannotator agreement measures. The scoring interface allow...

Description of the UPENN CAMP System as Used for Coreference

Scoring the performance of a system is an extremely important aspect of coreference algorithm performance. The score for a particular run is the single strongest measure of how well the system is performing and it can strongly determine directions for further improvements. In this paper, we present several di erent scoring algorithms and detail their respective strengths and weaknesses for vary...